Gesture Recognition and Time Series Representation
Ferhat Turhan
11/5/2021
In this homework, a multivariate time series data is tried to be visualized and understood. The data is a motion data expressed in three dimension: X,Y,Z. This data is actually for efficient personalized gesture recognition on wide range of devices. In this homework, the aim is only to make some manipulations on data and try to anticipate which class belongs to which gesture and after that finding representations to this time series.
In the first part manipulations are made and then some random motions are visualized so that some guesses about their classification on gestures could be made.
In the second part two alternative time series representations are proposed for this multivariate time series.
Part 1
Data Manipulation
First I read the data. Then to be able to make the this data more readable and for easier visualization, I did some manipulations such changing the first variables name as “class” and addind an “id” column. Since I will work on long format, I melted the data. After these, I used cumulative sum operation to find velocity and location information of the motion. These manipulations are applied to all of the three dimensions X,Y,Z.
## id class time acceleration_x velocity_x location_x
## 1: 1 1 1 -0.7914472 -0.7914472 -0.7914472
## 2: 1 1 2 -0.7914472 -1.5828944 -2.3743415
## 3: 1 1 3 -0.7958727 -2.3787671 -4.7531086
## 4: 1 1 4 -0.8100650 -3.1888321 -7.9419407
## 5: 1 1 5 -0.8492300 -4.0380621 -11.9800028
## 6: 1 1 6 -0.9034648 -4.9415269 -16.9215298## id class time acceleration_y velocity_y location_y
## 1: 1 1 1 -1.959984 -1.959984 -1.959984
## 2: 1 1 2 -1.959984 -3.919968 -5.879951
## 3: 1 1 3 -1.956600 -5.876568 -11.756519
## 4: 1 1 4 -1.945750 -7.822319 -19.578838
## 5: 1 1 5 -1.915809 -9.738128 -29.316966
## 6: 1 1 6 -1.874347 -11.612474 -40.929440## id class time acceleration_z velocity_z location_z
## 1: 1 1 1 -0.2490781 -0.2490781 -0.2490781
## 2: 1 1 2 -0.2490781 -0.4981562 -0.7472342
## 3: 1 1 3 -0.2514390 -0.7495951 -1.4968294
## 4: 1 1 4 -0.2590101 -1.0086052 -2.5054346
## 5: 1 1 5 -0.2799033 -1.2885085 -3.7939431
## 6: 1 1 6 -0.3088358 -1.5973444 -5.3912875Now that we are done with manipulations, my data is ready to be visualized.
To be able to see a motion’s shape, I need to visualize the motion in three dimensions in one diagram. Therefore I merged the location information of the three dimensions into one data table and then plot a couple of motions to see if they match to the motions described in the homework.
## id class time location_x location_y location_z
## 1: 1 1 1 -0.7914472 -1.959984 -0.2490781
## 2: 1 1 2 -2.3743415 -5.879951 -0.7472342
## 3: 1 1 3 -4.7531086 -11.756519 -1.4968294
## 4: 1 1 4 -7.9419407 -19.578838 -2.5054346
## 5: 1 1 5 -11.9800028 -29.316966 -3.7939431
## 6: 1 1 6 -16.9215298 -40.929440 -5.3912875Visualization
Let’s see for one random motion from each class and make guesses about them on which gesture they might fit in. (You can change the view of the plots by using left click and the mouse.)
This motion (ID 56) from “class 1” looks like a half circle. Along the gestures from homework, it can be said that this class might be gesture 7 or 8.
This motion (ID 124) from “class 2” looks like a wrongly finished square. Along the gestures from homework, it can be said that this class might be gesture 2, which is a square.
This motion (ID 310) from “class 3” looks like the letter “C”, which again made me think of the gesture 7 or 8 (which are full circles).
This motion (ID 455) from “class 4” looks like an unfinished shape of gesture 1.
This motion (ID 551) from “class 5” looks like straight line, which made me think of the gesture 3,4,5 or 6.
This motion (ID 643) from “class 6” looks like the letter “C”, which again made me think of the gesture 7 or 8 (which are full circles).
This motion (ID 792) from “class 7” is almost a full circle, which is probably gesture 7 or 8.
This motion (ID 844) from “class 8” made me think of changes in directions. But it also looks like a circle at the same time. I would say gesture 1,7,8 might be the classification.
Part 2
Representing as the Distance from Origin (0,0,0)
To be able to represent the data, I calculated the distances of all points to the origin point (0,0,0). Then I divided every motion into 2 part and calculated mean of these 2 intervals’ distance. After looking at the motions’ first and second intervals’ mean distances we can say that this is not a good representation for this data since there is no clear distinction between classes. But still when we look at the second intervals we might say that this is slightly better than the first interval in visual distinction.
Piecewise Aggregate Approximation
As the second approach, I used piecewise aggregate approximation. First for all classes I calculated the mean location values and then applied piecewise aggregate approximation to all dimensions.
By looking at all three dimensions, it can be seen that this second approach is much more better than the first approach when we try to distinguish class labels. Piecewise Aggregate Approximation definitely works well.
Conclusion
In this homework a multivariate time series data is examined and visualized. In the first part it can be seen that this time series actually represents the gestures and that is a huge step. We can see that images are hidden in these time series and they can be found and classified by data manipulation and some more advanced methods.
As in the second part some representations are tried to be made to be able to summarize data with few numbers. As mentioned above, “Piecewise Aggregate Approximation” is better than “Representing as the Distance from Origin (0,0,0)” From the graphs and plots, it is clear that we can distinguish the classes better in the second proposed method.